Note
This page corresponds to a Jupyter notebook you can
try out yourself.
(The original version is here.)
Tutorial¶
[1]:
# The plotting examples below require holoviews, hvplot, and bokeh:
# conda install -c conda-forge bokeh holoviews hvplot
import numpy as np
import pandas as pd
import bokeh
import hvplot.pandas
import holoviews as hv
import bokeh.palettes
from bokeh.plotting import figure, show, output_notebook
output_notebook()
Create a Client¶
Before you begin, you must create a Client object. It will be stored globally and used for all communication with the neuprint server.
Initialize it with your personal authentication token. See the Quickstart guide for details.
[2]:
from neuprint import Client
TOKEN = "" # <--- Paste your token here
# (or define NEUPRINT_APPLICATION CREDENTIALS in your environment)
c = Client('neuprint.janelia.org', 'hemibrain:v1.0.1', TOKEN)
ROIs¶
In neuprint, each neuron is annotated with the list of regions (ROIs) it intersects, along with the synapse counts in each.
The ROIs comprise a hierarchy, with smaller ROIs nested within larger ROIs. Furthermore, primary ROIs are guaranteed not to overlap, and they roughly tile the entire brain (with some gaps).
For a quick overview of the ROI hierarchy, use fetch_roi_hierarchy().
[3]:
from neuprint import fetch_roi_hierarchy
# Show the ROI hierarchy, with primary ROIs marked with '*'
print(fetch_roi_hierarchy(False, mark_primary=True, format='text'))
hemibrain
+-- AL(L)*
+-- AL(R)*
+-- AOT(R)
+-- CX
| +-- AB(L)*
| +-- AB(R)*
| +-- EB*
| +-- FB*
| +-- NO*
| +-- PB*
+-- GC
+-- GF(R)
+-- GNG*
+-- INP
| +-- ATL(L)*
| +-- ATL(R)*
| +-- CRE(L)*
| +-- CRE(R)*
| +-- IB*
| +-- ICL(L)*
| +-- ICL(R)*
| +-- SCL(L)*
| +-- SCL(R)*
+-- LH(R)*
+-- LX(L)
| +-- BU(L)*
| +-- LAL(L)*
+-- LX(R)
| +-- BU(R)*
| +-- LAL(R)*
+-- MB(+ACA)(R)
| +-- MB(R)
| | +-- CA(R)*
| | +-- PED(R)*
| | +-- a'L(R)*
| | +-- aL(R)*
| | +-- b'L(R)*
| | +-- bL(R)*
| | +-- gL(R)*
| +-- dACA(R)
| +-- lACA(R)
| +-- vACA(R)
+-- MB(L)
| +-- CA(L)*
| +-- a'L(L)*
| +-- aL(L)*
| +-- b'L(L)*
| +-- bL(L)*
| +-- gL(L)*
+-- OL(R)
| +-- AME(R)*
| +-- LO(R)*
| +-- LOP(R)*
| +-- ME(R)*
+-- PENP
| +-- CAN(R)*
| +-- FLA(R)*
| +-- PRW*
| +-- SAD*
+-- POC
+-- SNP(L)
| +-- SIP(L)*
| +-- SMP(L)*
+-- SNP(R)
| +-- SIP(R)*
| +-- SLP(R)*
| +-- SMP(R)*
+-- VLNP(R)
| +-- AOTU(R)*
| +-- AVLP(R)*
| +-- PLP(R)*
| +-- PVLP(R)*
| +-- WED(R)*
+-- VMNP
| +-- EPA(L)*
| +-- EPA(R)*
| +-- GOR(L)*
| +-- GOR(R)*
| +-- IPS(R)*
| +-- SPS(L)*
| +-- SPS(R)*
| +-- VES(L)*
| +-- VES(R)*
+-- mALT(L)
+-- mALT(R)
Neuron Search Criteria¶
Specify neurons of interest by bodyId, type/instance, or via a NeuronCriteria object. With NeuronCriteria, you can specify multiple search constraints, including the ROIs in which matched neurons must contain synapses.
[4]:
from neuprint import NeuronCriteria as NC
# Example: Select a specific body
criteria = 387023620
criteria = NC(bodyId=387023620)
# Example: Select several bodies
criteria = [387023620, 387364605, 416642425]
criteria = NC(bodyId=[387023620, 387364605, 416642425])
# Example: Select bodies by exact type
criteria = 'PEN_b(PEN2)'
criteria = NC(type='PENPEN_b(PEN2)')
# Example: Select bodies by exact instance
criteria = 'PEN(PB06)_b_L4'
criteria = NC(type='PEN(PB06)_b_L4')
# Example: Select bodies by type name pattern
criteria = NC(type='PEN.*', regex=True)
# Example: Select bodies by region (input or output)
criteria = NC(rois=['PB', 'EB'])
# Example: Select traced neurons which intersect the PB ROI with at least 100 inputs (PSDs).
criteria = NC(inputRois=['PB'], min_roi_inputs=100, status='Traced', cropped=False)
Fetch neuron properties¶
Neuron properties and per-ROI synapse distributions can be obtained with fetch_neurons(). Two dataframes are returned: one for neuron properties, and one for the counts of synapses in each ROI.
[5]:
from neuprint import fetch_neurons
neuron_df, roi_counts_df = fetch_neurons(criteria)
The total count of pre-synaptic and post-synaptic points within each neuron are given in the pre and post columns:
[6]:
neuron_df[['bodyId', 'instance', 'type', 'pre', 'post', 'status', 'cropped', 'size']]
[6]:
| bodyId | instance | type | pre | post | status | cropped | size | |
|---|---|---|---|---|---|---|---|---|
| 0 | 387023620 | PEN(PB06)_b_L4 | PEN_b(PEN2) | 343 | 1714 | Traced | False | 685261005 |
| 1 | 387364605 | EPG(PB08)_L3 | EPG | 645 | 4333 | Traced | False | 1459837829 |
| 2 | 416642425 | EPG(PB08)_R4 | EPG | 470 | 3828 | Traced | False | 1165123520 |
| 3 | 417998204 | PFNv(PB05)_R4 | PFNv | 229 | 990 | Traced | False | 473864774 |
| 4 | 419060083 | PFGs(PB10)_L2 | PFGs | 556 | 2395 | Traced | False | 1103881970 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 476 | 5813080979 | PEN(PB06)_a_L5 | PEN_a(PEN1) | 544 | 1595 | Traced | False | 924457147 |
| 477 | 5813081494 | PFR_b(PB11b)_R1 | PFR_b | 118 | 1891 | Traced | False | 588428634 |
| 478 | 5813081686 | PFNa(PB03)_L8 | PFNa | 121 | 510 | Traced | False | 283961924 |
| 479 | 5813087532 | IbSpsP(PB17)_R9 | IbSpsP | 108 | 388 | Traced | False | 257354274 |
| 480 | 5813128308 | PFNp(PB01)_a_L7 | PFNp_a | 71 | 310 | Traced | False | 187208043 |
481 rows × 8 columns
The per-ROI synapse counts are returned in the second DataFrame.
Note: Since ROIs overlap (see hierarchy above), the sum of the per-ROI counts for each body will be more than the pre and post columns above.
[7]:
roi_counts_df
[7]:
| bodyId | roi | pre | post | |
|---|---|---|---|---|
| 0 | 387023620 | CX | 341 | 1704 |
| 1 | 387023620 | PB | 1 | 777 |
| 2 | 387023620 | PB(L5) | 0 | 33 |
| 3 | 387023620 | PB(L4) | 1 | 723 |
| 4 | 387023620 | INP | 0 | 5 |
| ... | ... | ... | ... | ... |
| 7289 | 5813128308 | FB | 42 | 33 |
| 7290 | 5813128308 | FBl1 | 42 | 33 |
| 7291 | 5813128308 | NO | 27 | 107 |
| 7292 | 5813128308 | NO(R) | 27 | 107 |
| 7293 | 5813128308 | NO3(R) | 27 | 107 |
7294 rows × 4 columns
Fetch connections¶
Find synaptic connection strengths between one set of neurons and another using fetch_adjacencies().
The “source” and/or “target” neurons are selected using NeuronCriteria. Additional parameters allow you to filter by connection strength or ROI. Two DataFrames are returned, for neuron properties and per-ROI connection strengths.
[8]:
from neuprint import fetch_adjacencies, NeuronCriteria as NC
# Example: Fetch all downstream connections FROM a set of neurons
neuron_df, conn_df = fetch_adjacencies([387023620, 387364605, 416642425], None)
# Example: Fetch all upstream connections TO a set of neurons
neuron_df, conn_df = fetch_adjacencies(None, [387023620, 387364605, 416642425])
# Example: Fetch all direct connections between a set of upstream neurons and downstream neurons
neuron_df, conn_df = fetch_adjacencies(NC(type='Delta.*', regex=True), NC(type='PEN.*', regex=True))
[9]:
conn_df.sort_values('weight', ascending=False)
[9]:
| bodyId_pre | bodyId_post | roi | weight | |
|---|---|---|---|---|
| 179 | 910783961 | 5813070465 | PB | 184 |
| 57 | 880880259 | 849421763 | PB | 141 |
| 316 | 911565419 | 5813070465 | PB | 141 |
| 92 | 910442723 | 849421763 | PB | 139 |
| 216 | 911129204 | 724280817 | PB | 127 |
| ... | ... | ... | ... | ... |
| 351 | 911578496 | 724280817 | PB | 1 |
| 349 | 911578496 | 664645558 | PB | 1 |
| 348 | 911578496 | 634608104 | ATL(L) | 1 |
| 346 | 911578496 | 509410587 | PB | 1 |
| 660 | 5813061383 | 5813070465 | PB | 1 |
657 rows × 4 columns
Merge arbitrary neuron properties onto the connection table with merge_neuron_properties()
[10]:
from neuprint import merge_neuron_properties
conn_df = merge_neuron_properties(neuron_df, conn_df, ['type', 'instance'])
conn_df
[10]:
| bodyId_pre | bodyId_post | roi | weight | type_pre | instance_pre | type_post | instance_post | |
|---|---|---|---|---|---|---|---|---|
| 0 | 734581598 | 387023620 | PB | 1 | Delta7_a | Delta7(PB15)_a_L | PEN_b(PEN2) | PEN(PB06)_b_L4 |
| 1 | 734581598 | 508793049 | PB | 1 | Delta7_a | Delta7(PB15)_a_L | PEN_a(PEN1) | PEN(PB06)_a_L7 |
| 2 | 734581598 | 509410587 | PB | 3 | Delta7_a | Delta7(PB15)_a_L | PEN_a(PEN1) | PEN(PB06)_a_R4 |
| 3 | 734581598 | 569775058 | PB | 17 | Delta7_a | Delta7(PB15)_a_L | PEN_b(PEN2) | PEN(PB06)_b_R4 |
| 4 | 734581598 | 570461892 | PB | 21 | Delta7_a | Delta7(PB15)_a_L | PEN_a(PEN1) | PEN(PB06)_a_R3 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 652 | 5813061383 | 1508334312 | PB | 1 | Delta7_b | Delta7(PB15)_b_R | PEN_a(PEN1) | PEN(PB06)_a_L3 |
| 653 | 5813061383 | 1600083889 | PB | 37 | Delta7_b | Delta7(PB15)_b_R | PEN_b(PEN2) | PEN(PB06)_b_L9 |
| 654 | 5813061383 | 1601771527 | PB | 4 | Delta7_b | Delta7(PB15)_b_R | PEN_b(PEN2) | PEN(PB06)_b_R9 |
| 655 | 5813061383 | 5813047157 | PB | 12 | Delta7_b | Delta7(PB15)_b_R | PEN_b(PEN2) | PEN(PB06)_b_L2 |
| 656 | 5813061383 | 5813070465 | PB | 1 | Delta7_b | Delta7(PB15)_b_R | PEN_b(PEN2) | PEN(PB06)_b_L3 |
657 rows × 8 columns
Connection Matrix¶
You can convert a connection table into a connectivity matrix via connection_table_to_matrix().
[11]:
from neuprint.utils import connection_table_to_matrix
matrix = connection_table_to_matrix(conn_df, 'bodyId', sort_by='type')
matrix.iloc[:10, :10]
[11]:
| bodyId_post | 634608104 | 1508334312 | 664645558 | 509410587 | 5813056953 | 508793049 | 1125964630 | 910447075 | 849482511 | 847336755 |
|---|---|---|---|---|---|---|---|---|---|---|
| bodyId_pre | ||||||||||
| 850182049 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 1067331159 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 1129396529 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 755950242 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 1225601395 | 0 | 0 | 0 | 1 | 1 | 1 | 0 | 0 | 0 | 0 |
| 1009043211 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 946300031 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 911906936 | 20 | 13 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 15 |
| 911901802 | 0 | 4 | 0 | 0 | 1 | 0 | 0 | 0 | 33 | 17 |
| 911911699 | 37 | 16 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 |
[12]:
matrix.hvplot.heatmap(height=600, width=700).opts(xrotation=60)
[12]:
[13]:
matrix = connection_table_to_matrix(conn_df, ('bodyId_pre', 'type_post'))
matrix.hvplot.heatmap(height=600, width=400)
[13]:
Synapses¶
Fetch synapses for a set of bodies using NeuronCriteria, and optionally apply additional filtering with SynapseCriteria.
[14]:
from neuprint import fetch_synapses, NeuronCriteria as NC, SynapseCriteria as SC
neuron_criteria = NC(status='Traced', type='FB4Y', cropped=False, inputRois=['EB'], min_roi_inputs=100, min_pre=400)
eb_tbar_criteria = SC(rois='EB', type='pre', primary_only=True)
eb_tbars = fetch_synapses(neuron_criteria, eb_tbar_criteria)
[15]:
# Plot the synapse positions in a 2D projection
p = figure()
p.scatter(eb_tbars['x'], eb_tbars['z'])
p.y_range.flipped = True
show(p)
Synapse Connections¶
Here, we fetch all synapse-synapse connections from a set of neurons. Provide a NeuronCriteria for the source or target neurons (or both) to filter the neurons of interest, and optionally filter the synapses themselves via SynapseCriteria.
[16]:
from neuprint import fetch_synapse_connections
eb_syn_criteria = SC(rois='EB', primary_only=True)
eb_conns = fetch_synapse_connections(neuron_criteria, None, eb_syn_criteria)
eb_conns.head()
[16]:
| bodyId_pre | bodyId_post | roi_pre | roi_post | x_pre | y_pre | z_pre | x_post | y_post | z_post | confidence_pre | confidence_post | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 5813027016 | 978733459 | EB | EB | 26199 | 23565 | 18002 | 26210 | 23553 | 18003 | 0.977 | 0.947756 |
| 1 | 5813027016 | 978733459 | EB | EB | 25559 | 25038 | 23046 | 25554 | 25038 | 23056 | 0.928 | 0.920682 |
| 2 | 5813027016 | 978733459 | EB | EB | 23472 | 24677 | 21980 | 23464 | 24674 | 21970 | 0.683 | 0.692000 |
| 3 | 5813027016 | 978733459 | EB | EB | 28490 | 25331 | 22543 | 28519 | 25332 | 22545 | 0.956 | 0.752377 |
| 4 | 5813027016 | 978733459 | EB | EB | 29141 | 24584 | 19235 | 29129 | 24559 | 19240 | 0.953 | 0.334432 |
Let’s determine the post-synaptic neuron types, and plot the synapses for the top 5 types.
[17]:
# Retrieve the types of the post-synaptic neurons
post_neurons, _ = fetch_neurons(eb_conns['bodyId_post'].unique())
eb_conns = merge_neuron_properties(post_neurons, eb_conns, 'type')
top5_counts = eb_conns['type_post'].value_counts().head(5)
top5_counts
[17]:
EPG 1480
EQ5 571
PEN_a(PEN1) 374
ExR7 264
ExR4 230
Name: type_post, dtype: int64
[18]:
colormap = dict(zip(top5_counts.index, bokeh.palettes.Category10[5]))
points = eb_conns.query('type_post in @top5_counts.index').copy()
points['color'] = points['type_post'].map(colormap)
p = figure()
p.scatter(points['x_post'], points['z_post'], color=points['color'])
p.y_range.flipped = True
show(p)
Skeletons¶
Download skeletons with Client.fetch_skeleton().
[19]:
# Download some skeletons as DataFrames and attach columns for bodyId and color
skeletons = []
for i, bodyId in enumerate(eb_conns['bodyId_pre'].unique()):
s = c.fetch_skeleton(bodyId, format='pandas')
s['bodyId'] = bodyId
s['color'] = bokeh.palettes.Accent[5][i]
skeletons.append(s)
# Combine into one big table for convenient processing
skeletons = pd.concat(skeletons, ignore_index=True)
skeletons.head()
[19]:
| rowId | x | y | z | radius | link | bodyId | color | |
|---|---|---|---|---|---|---|---|---|
| 0 | 1 | 21626.0 | 34402.0 | 12790.0 | 10.0000 | -1 | 5813027016 | #7fc97f |
| 1 | 2 | 21626.0 | 34402.0 | 12810.0 | 10.0000 | 1 | 5813027016 | #7fc97f |
| 2 | 3 | 21626.0 | 34382.0 | 12830.0 | 10.0000 | 2 | 5813027016 | #7fc97f |
| 3 | 4 | 21626.0 | 34362.0 | 12850.0 | 18.2843 | 3 | 5813027016 | #7fc97f |
| 4 | 5 | 21606.0 | 34342.0 | 12870.0 | 18.2843 | 4 | 5813027016 | #7fc97f |
[20]:
# Join parent/child nodes for plotting as line segments below.
segments = skeletons.merge(skeletons, 'inner',
left_on=['bodyId', 'rowId'],
right_on=['bodyId', 'link'],
suffixes=['_child', '_parent'])
[21]:
p = figure()
p.y_range.flipped = True
# Plot skeleton segments (in 2D)
p.segment(x0='x_child', x1='x_parent',
y0='z_child', y1='z_parent',
color='color_child',
source=segments)
# Also plot the synapses from the above example
p.scatter(points['x_post'], points['z_post'], color=points['color'])
show(p)